1. Field of the Invention
The present invention relates to image combining devices that combine a plurality of images and to image combining methods for combining a plurality of images. The present invention also relates to imaging apparatuses that adopt such image combining devices or image combining methods.
2. Description of Related Art
In recent years, an image stabilization technique, which is a technique for reducing camera shake when shooting with a camera, has been actively studied. Irrespective of whether a stabilization target is a still image or a moving image, the image stabilization technique can be broken down into a technique for detecting camera shake and a technique for stabilizing an image based on the detection result.
As a method for detecting camera shake, there are a method that uses a sensor (an angular speed sensor) and an electronic method that analyzes an image for detection. As for image stabilization, there are optical image stabilization by which an image is stabilized by the driving of an optical system and electronic image stabilization by which an image is stabilized by image processing.
As a method for performing image stabilization for a still image (or a moving image), there are a method by which camera shake is detected with a sensor so that optical image stabilization is performed based on the detection result, a method by which camera shake is detected with a sensor so that electronic image stabilization is performed based on the detection result, and a method by which camera shake is detected by an image analysis so that electronic image stabilization is performed based on the detection result.
As a method by which camera shake is detected by an image analysis so that electronic image stabilization is performed based on the detection result, image stabilization by image addition has come into practical use. The image stabilization by image addition is performed as follows. A plurality of divided exposure images (short-time exposure images) G1 to G4 exposed for an exposure time T2 obtained by dividing a normal exposure time T1 are positioned in such a way as to cancel a difference between them, and are then combined together one on top of another. In this way, one still image that suffers less from camera shake is formed (see FIG. 24).
As a method for detecting a difference (in other words, a motion) between images by image processing, there are an image matching method, a feature point matching method, a gradient method, and the like. The image matching method is performed as follows. Two images are placed one on top of another and are gradually shifted from each other, so as to find out a position (a relative positional difference between them) where the smallest difference is observed between the two images. In this case, frequently used as a value representing the difference between the two images is the sum of absolute difference (SAD) or the sum of squared difference (SSD) between the pixel values of these two images.
The image matching method is classified into a pixel-by-pixel matching method (a block matching method) and a representative point matching method. By the pixel-by-pixel matching method, the difference between two images to be compared is calculated by using the pixel values of all pixels present in a portion where the two images overlap each other. On the other hand, by the representative point matching method, the difference between two images is calculated by using a pixel value of a representative point that is set at intervals of several to several tens of pixels. Since the representative point matching method allows easy implementation by hardware and can realize practical detection accuracy, it is widely used in imaging apparatuses such as digital video cameras.
Conventional image stabilization by image addition is performed as follows. A plurality of divided exposure images are individually positioned based on the image features thereof with reference to a reference image (typically, the first divided exposure image) selected from among the plurality of divided exposure images, and are then simply added together to form one still image. FIG. 25 shows a typical example of how a plurality of divided exposure images G1 to G4 are combined together to form one still image as a composite image. It is to be noted that FIG. 25 shows a case in which all the divided exposure images suffer less from image blur.
In general, the exposure time of each divided exposure image is set to be short enough to make camera shake negligible. However, a plurality of divided exposure images may include a divided exposure image with significant image blur caused by camera shake or moving subject. In such a case, the divided exposure images are positioned with a lower degree of accuracy. As a result, as shown in FIG. 26, a still image obtained as a composite image blurs (in FIG. 26, there is significant blur in the divided exposure image G4).
In addition, the first divided exposure image, for example, of a plurality of divided exposure images is set as a reference image, and the plurality of divided exposure images are then positioned with reference to the reference image thus set. However, the first divided exposure image fixed as a reference image is not always the most suitable reference image. The reason is as follows. The positioning is performed by converting the image coordinates of the divided exposure images other than the reference image into an image coordinate system of the reference image. This coordinate transformation generally entails interpolation of pixel values, resulting in some image degradation in the divided exposure images other than the reference image. This makes it preferable to select a reference image that helps minimize such image degradation.